{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2020 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/samples/tensorflow/sentiment_analysis/ai_platform_sentiment_analysis.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/samples/tensorflow/sentiment_analysis/ai_platform_sentiment_analysis.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Yy8l-95JkRx0"
   },
   "source": [
    "# Overview\n",
    "\n",
    "AI Platform Online Prediction now supports custom python code in to apply custom prediction routines, in this blog post we will perform sentiment analysis using [Twitter data](https://www.kaggle.com/kazanova/sentiment140) and Transfer learning using [Pretrained Glove embeddings](https://nlp.stanford.edu/projects/glove/). This tutorial also uses the new [AI Platform Pipelines](https://cloud.google.com/ai-platform/pipelines/docs) product.\n",
    "\n",
    "\n",
    "### Dataset\n",
    "\n",
    "We use the [Twitter data](https://www.kaggle.com/kazanova/sentiment140) which is called `sentiment140` dataset. It contains 1,600,000 tweets extracted using the Twitter AI. The tweets have been annotated (0 = negative, 4 = positive) and they can be used to detect sentiment.\n",
    "\n",
    "It contains the following 6 fields:\n",
    "\n",
    "```\n",
    "target: the polarity of the tweet (0 = negative, 2 = neutral, 4 = positive)\n",
    "ids: The id of the tweet ( 2087)\n",
    "date: the date of the tweet (Sat May 16 23:58:44 UTC 2009)\n",
    "flag: The query (lyx). If there is no query, then this value is NO_QUERY.\n",
    "user: the user that tweeted (robotickilldozr)\n",
    "text: the text of the tweet (Lyx is cool)\n",
    "```\n",
    "The official link regarding the dataset with resources about how it was generated is [here](http://%20http//help.sentiment140.com/for-students/)\n",
    "\n",
    "### Objective\n",
    "\n",
    "In this notebook, we show how to deploy a TensorFlow model using AI Platform  Custom Prediction Code using `sentiment140` for sentiment analysis.\n",
    "\n",
    "### Costs \n",
    "\n",
    "This tutorial uses billable components of Google Cloud Platform (GCP):\n",
    "\n",
    "* Cloud AI Platform\n",
    "* Cloud Storage\n",
    "\n",
    "Learn about [Cloud AI Platform\n",
    "pricing](https://cloud.google.com/ml-engine/docs/pricing) and [Cloud Storage\n",
    "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
    "Calculator](https://cloud.google.com/products/calculator/)\n",
    "to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D5AirRX0kRx2"
   },
   "source": [
    "### Set up your local development environment\n",
    "\n",
    "**If you are using Colab or AI Platform Notebooks**, your environment already meets\n",
    "all the requirements to run this notebook. You can skip this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
    "You need the following:\n",
    "\n",
    "* The Google Cloud SDK\n",
    "* Git\n",
    "* Python 3\n",
    "* virtualenv\n",
    "* Jupyter notebook running in a virtual environment with Python 3\n",
    "\n",
    "The Google Cloud guide to [Setting up a Python development\n",
    "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
    "installation guide](https://jupyter.org/install) provide detailed instructions\n",
    "for meeting these requirements. The following steps provide a condensed set of\n",
    "instructions:\n",
    "\n",
    "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
    "\n",
    "2. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
    "\n",
    "3. [Install\n",
    "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
    "   and create a virtual environment that uses Python 3.\n",
    "\n",
    "4. Activate that environment and run `pip install jupyter` in a shell to install\n",
    "   Jupyter.\n",
    "\n",
    "5. Run `jupyter notebook` in a shell to launch Jupyter.\n",
    "\n",
    "6. Open this notebook in the Jupyter Notebook Dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up your GCP project\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
    "\n",
    "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
    "\n",
    "3. [Enable the AI Platform APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n",
    "\n",
    "4. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
    "Cloud SDK uses the right project for all the commands in this notebook.\n",
    "\n",
    "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate your GCP account\n",
    "\n",
    "**If you are using AI Platform Notebooks**, your environment is already\n",
    "authenticated. Skip this step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**If you are using Colab**, run the cell below and follow the instructions\n",
    "when prompted to authenticate your account via oAuth.\n",
    "\n",
    "**Otherwise**, follow these steps:\n",
    "\n",
    "1. In the GCP Console, go to the [**Create service account key**\n",
    "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
    "\n",
    "2. From the **Service account** drop-down list, select **New service account**.\n",
    "\n",
    "3. In the **Service account name** field, enter a name.\n",
    "\n",
    "4. From the **Role** drop-down list, select\n",
    "   **Machine Learning Engine > AI Platform Admin** and\n",
    "   **Storage > Storage Object Admin**.\n",
    "\n",
    "5. Click *Create*. A JSON file that contains your key downloads to your\n",
    "local environment.\n",
    "\n",
    "6. Enter the path to your service account key as the\n",
    "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are running this notebook in Colab, run the following cell to authenticate your Google Cloud Platform user account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "# If you are running this notebook in Colab, run this cell and follow the\n",
    "# instructions to authenticate your GCP account. This provides access to your\n",
    "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
    "# requests.\n",
    "\n",
    "if 'google.colab' in sys.modules:\n",
    "  from google.colab import auth as google_auth\n",
    "  google_auth.authenticate_user()\n",
    "\n",
    "# If you are running this notebook locally, replace the string below with the\n",
    "# path to your service account key and run this cell to authenticate your GCP\n",
    "# account.\n",
    "else:\n",
    "  %env GOOGLE_APPLICATION_CREDENTIALS ''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PIP Install Packages and dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U tensorflow==1.15.* --user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Project Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_ID = '[your-project-id]' # TODO (Set to your GCP Project name)\n",
    "!gcloud config set project {PROJECT_ID}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME = '[your-bucket-name]' # TODO (Set to your GCS Bucket name)\n",
    "REGION = 'us-central1' #@param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model information.\n",
    "ROOT = 'ml_pipeline'\n",
    "MODEL_DIR = os.path.join(ROOT,'models').replace(\"\\\\\",\"/\")\n",
    "PACKAGES_DIR = os.path.join(ROOT,'packages').replace(\"\\\\\",\"/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "!gsutil rm -r gs://{BUCKET_NAME}/{ROOT}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Get training data\n",
    "\n",
    "In this step, we are going to:\n",
    "1. Download Twitter data\n",
    "2. Load the data to Pandas Dataframe.\n",
    "3. Convert the class feature (sentiment) from string to a numeric indicator.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data can be downloaded directly from [here](https://www.kaggle.com/kazanova/sentiment140) (https://www.kaggle.com/kazanova/sentiment140)\n",
    "\n",
    "It is also located here: `gs://cloud-samples-data/ai-platform/sentiment_analysis/training.csv`\n",
    "\n",
    "You can copy it by using the following command:\n",
    "\n",
    "```\n",
    "gsutil cp gs://cloud-samples-data/ai-platform/sentiment_analysis/training.csv .\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gsutil cp gs://cloud-samples-data/ai-platform/sentiment_analysis/training.csv ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Input data\n",
    "\n",
    "Create a dictionary with a mapping for each label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentiment_mapping = {\n",
    "    0: 'negative',\n",
    "    2: 'neutral',\n",
    "    4: 'positive'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "df_twitter = pd.read_csv('training.csv', encoding='latin1', header=None)\\\n",
    "             .rename(columns={\n",
    "                 0: 'sentiment',\n",
    "                 1: 'id',\n",
    "                 2: 'posted_at',\n",
    "                 3: 'query',\n",
    "                 4: 'username',\n",
    "                 5: 'text'\n",
    "             })[['sentiment', 'text']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_twitter['sentiment_label'] = df_twitter['sentiment'].map(sentiment_mapping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify number of records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_twitter['sentiment_label'].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Data processing fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%writefile preprocess.py\n",
    "\n",
    "from tensorflow.python.keras.preprocessing import sequence\n",
    "from tensorflow.keras.preprocessing import text\n",
    "import re\n",
    "\n",
    "\n",
    "class TextPreprocessor(object):\n",
    "    def __init__(self, vocab_size, max_sequence_length):\n",
    "        self._vocab_size = vocab_size\n",
    "        self._max_sequence_length = max_sequence_length\n",
    "        self._tokenizer = None\n",
    "\n",
    "    def _clean_line(self, text):\n",
    "        text = re.sub(r\"http\\S+\", \"\", text)\n",
    "        text = re.sub(r\"@[A-Za-z0-9]+\", \"\", text)\n",
    "        text = re.sub(r\"#[A-Za-z0-9]+\", \"\", text)\n",
    "        text = text.replace(\"RT\",\"\")\n",
    "        text = text.lower()\n",
    "        text = text.strip()\n",
    "        return text\n",
    "    \n",
    "    def fit(self, text_list):        \n",
    "        # Create vocabulary from input corpus.\n",
    "        text_list_cleaned = [self._clean_line(txt) for txt in text_list]\n",
    "        tokenizer = text.Tokenizer(num_words=self._vocab_size)\n",
    "        tokenizer.fit_on_texts(text_list)\n",
    "        self._tokenizer = tokenizer\n",
    "\n",
    "    def transform(self, text_list):        \n",
    "        # Transform text to sequence of integers\n",
    "        text_list = [self._clean_line(txt) for txt in text_list]\n",
    "        text_sequence = self._tokenizer.texts_to_sequences(text_list)\n",
    "\n",
    "        # Fix sequence length to max value. Sequences shorter than the length are\n",
    "        # padded in the beginning and sequences longer are truncated\n",
    "        # at the beginning.\n",
    "        padded_text_sequence = sequence.pad_sequences(\n",
    "          text_sequence, maxlen=self._max_sequence_length)\n",
    "        return padded_text_sequence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some small test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from preprocess import TextPreprocessor\n",
    "\n",
    "processor = TextPreprocessor(5, 5)\n",
    "processor.fit(['hello Google Cloud AI Platform','test'])\n",
    "processor.transform(['hello Google Cloud AI Platform',\"lol\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLASSES = {'negative': 0, 'positive': 1}  # label-to-int mapping\n",
    "VOCAB_SIZE = 25000  # Limit on the number vocabulary size used for tokenization\n",
    "MAX_SEQUENCE_LENGTH = 50  # Sentences will be truncated/padded to this length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from preprocess import TextPreprocessor\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "sents = df_twitter.text\n",
    "labels = np.array(df_twitter.sentiment_label.map(CLASSES))\n",
    "\n",
    "# Train and test split\n",
    "X, _, y, _ = train_test_split(sents, labels, test_size=0.1)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)\n",
    "\n",
    "# Create vocabulary from training corpus.\n",
    "processor = TextPreprocessor(VOCAB_SIZE, MAX_SEQUENCE_LENGTH)\n",
    "processor.fit(X_train)\n",
    "\n",
    "# Preprocess the data\n",
    "train_texts_vectorized = processor.transform(X_train)\n",
    "eval_texts_vectorized = processor.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "with open('./processor_state.pkl', 'wb') as f:\n",
    "    pickle.dump(processor, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Model\n",
    "\n",
    "Create a TensorFlow model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperparameters\n",
    "\n",
    "LEARNING_RATE = .001\n",
    "EMBEDDING_DIM = 50\n",
    "FILTERS = 64\n",
    "DROPOUT_RATE = 0.5\n",
    "POOL_SIZE = 3\n",
    "NUM_EPOCH = 25\n",
    "BATCH_SIZE = 128\n",
    "KERNEL_SIZES = [2, 5, 8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Basic model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def create_model(vocab_size, embedding_dim, filters, kernel_sizes, dropout_rate, pool_size, embedding_matrix):\n",
    "    \n",
    "    # Input layer\n",
    "    model_input = tf.keras.layers.Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "\n",
    "    # Embedding layer\n",
    "    z = tf.keras.layers.Embedding(\n",
    "        input_dim=vocab_size + 1,\n",
    "        output_dim=embedding_dim,\n",
    "        input_length=MAX_SEQUENCE_LENGTH,\n",
    "        weights=[embedding_matrix]\n",
    "    )(model_input)\n",
    "\n",
    "    z = tf.keras.layers.Dropout(dropout_rate)(z)\n",
    "\n",
    "    # Convolutional block\n",
    "    conv_blocks = []\n",
    "    for kernel_size in kernel_sizes:\n",
    "        conv = tf.keras.layers.Convolution1D(\n",
    "            filters=filters,\n",
    "            kernel_size=kernel_size,\n",
    "            padding='valid',\n",
    "            activation='relu',\n",
    "            bias_initializer='random_uniform',\n",
    "            strides=1)(z)\n",
    "        conv = tf.keras.layers.MaxPooling1D(pool_size=2)(conv)\n",
    "        conv = tf.keras.layers.Flatten()(conv)\n",
    "        conv_blocks.append(conv)\n",
    "        \n",
    "    z = tf.keras.layers.Concatenate()(conv_blocks) if len(conv_blocks) > 1 else conv_blocks[0]\n",
    "\n",
    "    z = tf.keras.layers.Dropout(dropout_rate)(z)\n",
    "    z = tf.keras.layers.Dense(100, activation='relu')(z)\n",
    "    model_output = tf.keras.layers.Dense(1, activation='sigmoid')(z)\n",
    "    model = tf.keras.models.Model(model_input, model_output)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Pretrained Glove embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Embeddings can be downloaded from Stanford Glove project: https://nlp.stanford.edu/projects/glove/\n",
    "- Download file [here](http://nlp.stanford.edu/data/glove.twitter.27B.zip) (http://nlp.stanford.edu/data/glove.twitter.27B.zip)\n",
    "- Twitter (2B tweets, 27B tokens, 1.2M vocab, uncased, 25d, 50d, 100d, & 200d vectors, 1.42 GB download)\n",
    "\n",
    "It is also located here: `gs://cloud-samples-data/ai-platform/sentiment_analysis/glove.twitter.27B.50d.txt`\n",
    "\n",
    "You can copy it by using the following command:\n",
    "\n",
    "```\n",
    "gsutil cp gs://cloud-samples-data/ai-platform/sentiment_analysis/glove.twitter.27B.50d.txt .\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gsutil cp gs://cloud-samples-data/ai-platform/sentiment_analysis/glove.twitter.27B.50d.txt ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an embedding index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_coefs(word, *arr):\n",
    "    return word, np.asarray(arr, dtype='float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings_index = dict(get_coefs(*o.strip().split()) for o in open('glove.twitter.27B.50d.txt','r', encoding='utf8'))                                                                                                                                    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "word_index = processor._tokenizer.word_index\n",
    "nb_words = min(VOCAB_SIZE, len(word_index))\n",
    "embedding_matrix = np.zeros((nb_words + 1, EMBEDDING_DIM))\n",
    "\n",
    "for word, i in word_index.items():\n",
    "    if i >= VOCAB_SIZE: continue\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None: embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3. Create, compile and train TensorFlow model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "model = create_model(VOCAB_SIZE, EMBEDDING_DIM, FILTERS, KERNEL_SIZES, DROPOUT_RATE,POOL_SIZE, embedding_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile model with learning parameters.\n",
    "\n",
    "optimizer = tf.keras.optimizers.Nadam(lr=0.001)\n",
    "model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "#Keras train\n",
    "\n",
    "history = model.fit(\n",
    "    train_texts_vectorized, \n",
    "    y_train, \n",
    "    epochs=NUM_EPOCH, \n",
    "    batch_size=BATCH_SIZE,\n",
    "    validation_data=(eval_texts_vectorized, y_test),\n",
    "    verbose=2,\n",
    "    callbacks=[\n",
    "        tf.keras.callbacks.ReduceLROnPlateau(\n",
    "            monitor='val_acc',\n",
    "            min_delta=0.005,\n",
    "            patience=3,\n",
    "            factor=0.5),\n",
    "        tf.keras.callbacks.EarlyStopping(\n",
    "            monitor='val_loss',\n",
    "            min_delta=0.005, \n",
    "            patience=5, \n",
    "            verbose=0, \n",
    "            mode='auto'\n",
    "        ),\n",
    "        tf.keras.callbacks.History()\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('history.pkl','wb') as file:\n",
    "    pickle.dump(history.history,file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('keras_saved_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1. Prepare custom model prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%writefile custom_prediction.py\n",
    "\n",
    "import os\n",
    "import pickle\n",
    "import numpy as np\n",
    "\n",
    "from datetime import date\n",
    "from google.cloud import logging\n",
    "\n",
    "import tensorflow.keras as keras\n",
    "\n",
    "\n",
    "class CustomModelPrediction(object):   \n",
    "    def __init__(self, model, processor): \n",
    "        self._model = model\n",
    "        self._processor = processor       \n",
    "    \n",
    "    def _postprocess(self, predictions):\n",
    "        labels = ['negative', 'positive']\n",
    "        return [\n",
    "            {\n",
    "                \"label\":labels[int(np.round(prediction))],\n",
    "                \"score\":float(np.round(prediction, 4))\n",
    "            } for prediction in predictions]\n",
    "    \n",
    "    def predict(self, instances, **kwargs):       \n",
    "        preprocessed_data = self._processor.transform(instances)\n",
    "        predictions =  self._model.predict(preprocessed_data)\n",
    "        labels = self._postprocess(predictions)\n",
    "        return labels\n",
    "\n",
    "    @classmethod\n",
    "    def from_path(cls, model_dir):       \n",
    "        model = keras.models.load_model(\n",
    "          os.path.join(model_dir,'keras_saved_model.h5'))\n",
    "        with open(os.path.join(model_dir, 'processor_state.pkl'), 'rb') as f:\n",
    "            processor = pickle.load(f)           \n",
    "        return cls(model, processor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing custom prediction locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from custom_prediction import CustomModelPrediction\n",
    "classifier = CustomModelPrediction.from_path('.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "requests = (['God I hate the north', 'god I love this'])\n",
    "response = classifier.predict(requests)\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2. Package it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%writefile setup.py\n",
    "\n",
    "from setuptools import setup\n",
    "\n",
    "setup(\n",
    "  name='tweet_sentiment_classifier',\n",
    "  version='0.1',\n",
    "  include_package_data=True,\n",
    "  scripts=['preprocess.py', 'custom_prediction.py']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wrap it up and copy to GCP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "!python setup.py sdist\n",
    "!gsutil cp ./dist/tweet_sentiment_classifier-0.1.tar.gz gs://{BUCKET_NAME}/{PACKAGES_DIR}/tweet_sentiment_classifier-0.1.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "!gsutil cp keras_saved_model.h5 gs://{BUCKET_NAME}/{MODEL_DIR}/\n",
    "!gsutil cp processor_state.pkl gs://{BUCKET_NAME}/{MODEL_DIR}/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Create model and version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_NAME='twitter_model_custom_prediction'\n",
    "MODEL_VERSION='v1'\n",
    "\n",
    "RUNTIME_VERSION='1.15'\n",
    "PYTHON_VERSION='3.7'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "!gcloud beta ai-platform models create {MODEL_NAME} --regions {REGION}  --enable-logging  --enable-console-logging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "!gcloud ai-platform versions delete {MODEL_VERSION} --model {MODEL_NAME} --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "!gcloud beta ai-platform versions create {MODEL_VERSION} \\\n",
    "--model {MODEL_NAME} \\\n",
    "--origin gs://{BUCKET_NAME}/{MODEL_DIR} \\\n",
    "--python-version {PYTHON_VERSION} \\\n",
    "--runtime-version {RUNTIME_VERSION} \\\n",
    "--package-uris gs://{BUCKET_NAME}/{PACKAGES_DIR}/tweet_sentiment_classifier-0.1.tar.gz \\\n",
    "--prediction-class=custom_prediction.CustomModelPrediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from googleapiclient import discovery\n",
    "from oauth2client.client import GoogleCredentials\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "requests = [\n",
    "    'god this episode is bad',\n",
    "    'meh, I kinda like it',\n",
    "    'what were the writer thinking, omg!',\n",
    "    'omg! what a twist, who would\\'ve though :o!',\n",
    "    'woohoow, sansa for the win!'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# JSON format the requests\n",
    "request_data = {'instances': requests}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authenticate and call AI Plaform prediction API "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "api = discovery.build('ml', 'v1')\n",
    "parent = 'projects/{}/models/{}/versions/{}'.format(PROJECT_ID, MODEL_NAME, MODEL_VERSION)\n",
    "parent = 'projects/{}/models/{}'.format(PROJECT_ID, MODEL_NAME)\n",
    "response = api.projects().predict(body=request_data, name=parent).execute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "response['predictions']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete model version resource\n",
    "! gcloud ai-platform versions delete {MODEL_VERSION} --model {MODEL_NAME} --quiet\n",
    "\n",
    "# Delete model resource\n",
    "! gcloud ai-platform models delete {MODEL_NAME} --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Deploy using AI Platform Pipelines\n",
    "\n",
    "With AI Platform Pipelines, you can orchestrate your machine learning (ML) workflows as reusable and reproducible pipelines. AI Platform Pipelines saves you the difficulty of setting up Kubeflow Pipelines with TensorFlow Extended on Google Kubernetes Engine.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the KubeFlow Pipelines SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "!pip install 'kfp>=0.1.31' --user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "import kfp\n",
    "import kfp.components as comp\n",
    "import kfp.dsl as dsl\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Hosted AI Platform Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new Hosted KubeFlow pipeline under AI Platform -> Pipelines.\n",
    "\n",
    "Set up you AI Platform Pipeline as indicated [here](https://cloud.google.com/ai-platform/pipelines/docs/setting-up)\n",
    "\n",
    "**Note:** Verify you are using version 0.2.5 and above.\n",
    "More information [here](https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seting up credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run pipelines that requires calling any GCP services, such as Cloud Storage, Cloud ML Engine, Dataflow, or Dataproc, you need to set the application default credential to a pipeline step by mounting the proper GCP service account token as a Kubernetes secret.\n",
    "\n",
    "[Documentation here](https://github.com/kubeflow/pipelines/blob/master/manifests/gcp_marketplace/guide.md#gcp-service-account-credentials)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and deploy the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "# Project parameters.\n",
    "CLUSTER='' # TODO Change to your GKE cluster\n",
    "ZONE='us-central1-a'\n",
    "\n",
    "# Pipeline Parameters\n",
    "\n",
    "MODEL_NAME = 'sentiment_classifier' + str(int(time.time()))\n",
    "MODEL_VERSION = 'v1' + str(int(time.time()))\n",
    "RUNTIME_VERSION = '1.15'\n",
    "PYTHON_VERSION='3.7'\n",
    "\n",
    "PACKAGE_TRAINER_URI = 'gs://cloud-samples-data/ai-platform/sentiment_analysis/trainer-0.1.tar.gz'\n",
    "PACKAGE_CUSTOM_PREDICTION_URI = 'gs://cloud-samples-data/ai-platform/sentiment_analysis/custom_prediction-0.1.tar.gz'\n",
    "PACKAGE_URIS = json.dumps([PACKAGE_TRAINER_URI])\n",
    "PACKAGE_PATH='./trainer'\n",
    "PYTHON_MODULE = 'trainer.task'\n",
    "\n",
    "TRAINING_FILE='gs://cloud-samples-data/ai-platform/sentiment_analysis/training.csv'.format(BUCKET_NAME)\n",
    "GLOVE_FILE='gs://cloud-samples-data/ai-platform/sentiment_analysis/glove.twitter.27B.50d.txt'.format(BUCKET_NAME)\n",
    "\n",
    "MODEL_DIR='gs://{}/models'.format(BUCKET_NAME)\n",
    "\n",
    "SAVED_MODEL_NAME='keras_saved_model.h5'\n",
    "PROCESSOR_STATE_FILE='processor_state.pkl'\n",
    "\n",
    "\n",
    "PIPELINE_NAME = 'Text Prediction'\n",
    "PIPELINE_FILENAME_PREFIX = 'twitter'\n",
    "PIPELINE_DESCRIPTION = 'Text Prediction'\n",
    "\n",
    "\n",
    "# Note, numeric parameters should be pass as string.\n",
    "TRAINER_ARGS = json.dumps(['--train-file', TRAINING_FILE,\n",
    "                            '--glove-file', GLOVE_FILE,\n",
    "                            '--learning-rate', '0.001',\n",
    "                            '--embedding-dim', '50',\n",
    "                            '--num-epochs', '25',\n",
    "                            '--filter-size', '64',\n",
    "                            '--batch-size', '128',\n",
    "                            '--vocab-size', '25000',\n",
    "                            '--pool-size', '3',\n",
    "                            '--max-sequence-length', '50',\n",
    "                            '--saved-model', SAVED_MODEL_NAME,\n",
    "                            '--preprocessor-state-file', PROCESSOR_STATE_FILE,\n",
    "                            '--gcs-bucket', BUCKET_NAME,\n",
    "                            '--deploy-gcp']\n",
    "                       )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "aiplatform_train_op = comp.load_component_from_url(\n",
    "    'https://raw.githubusercontent.com/kubeflow/pipelines/master/components/gcp/ml_engine/train/component.yaml')\n",
    "\n",
    "def train(project_id,  trainer_args, package_uris, job_dir, region, python_module, python_version, runtime_version):\n",
    "    return aiplatform_train_op(\n",
    "        project_id=project_id,\n",
    "        python_module=python_module,\n",
    "        python_version=python_version,\n",
    "        package_uris=package_uris,\n",
    "        region=region,\n",
    "        args=trainer_args,\n",
    "        job_dir=job_dir,\n",
    "        runtime_version=runtime_version\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "aiplatform_deploy_op = comp.load_component_from_url(\n",
    "    'https://raw.githubusercontent.com/kubeflow/pipelines/master/components/gcp/ml_engine/deploy/component.yaml')\n",
    "\n",
    "def deploy(project_id, model_uri, model_id, model_version, runtime_version, python_version, version):\n",
    "    return aiplatform_deploy_op(\n",
    "        model_uri=model_uri,\n",
    "        project_id=project_id, \n",
    "        model_id=model_id, \n",
    "        version_id=model_version, \n",
    "        runtime_version=runtime_version,\n",
    "        python_version=python_version,\n",
    "        version=version,\n",
    "        replace_existing_version=True,       \n",
    "        set_default=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "@dsl.pipeline(\n",
    "    name=PIPELINE_NAME,\n",
    "    description=PIPELINE_DESCRIPTION\n",
    ")\n",
    "def pipeline(project_id=PROJECT_ID, \n",
    "             python_module=PYTHON_MODULE,\n",
    "             region=REGION,\n",
    "             runtime_version=RUNTIME_VERSION, \n",
    "             package_uris=PACKAGE_URIS,\n",
    "             python_version=PYTHON_VERSION,\n",
    "             job_dir=MODEL_DIR,):\n",
    "    train_task = train(project_id,\n",
    "                       TRAINER_ARGS,\n",
    "                       package_uris,\n",
    "                       job_dir,                      \n",
    "                       region,\n",
    "                       python_module,\n",
    "                       python_version,\n",
    "                       runtime_version)\n",
    "    \n",
    "    deploy_task = deploy(project_id,\n",
    "                         train_task.outputs['job_dir'],                         \n",
    "                         MODEL_NAME,\n",
    "                         MODEL_VERSION,\n",
    "                         runtime_version,\n",
    "                         python_version,\n",
    "                         {                           \n",
    "                          \"deploymentUri\": 'gs://news-ml/models', \n",
    "                          \"packageUris\": [PACKAGE_CUSTOM_PREDICTION_URI, PACKAGE_TRAINER_URI],\n",
    "                          \"predictionClass\": 'model_prediction.CustomModelPrediction'\n",
    "                         }\n",
    "                        )    \n",
    "    return True\n",
    "\n",
    "# Reference for invocation later\n",
    "pipeline_func = pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the KFP pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "!kubectl get secrets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "!gcloud container clusters get-credentials \"$CLUSTER\" --zone \"$ZONE\" --project \"$PROJECT_ID\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obtain the `KFP_HOST` variable from the AI Platform Managed pipelines screen in Google Cloud Console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "KFP_HOST = ''\n",
    "pipeline = kfp.Client(host=KFP_HOST).create_run_from_pipeline_func(pipeline, arguments={})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "no_execute"
    ]
   },
   "outputs": [],
   "source": [
    "pipeline.wait_for_run_completion(timeout=1800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**References:** \n",
    "\n",
    "- [AI in Depth: Serving a PyTorch text classifier on AI Platform Serving using custom online prediction](https://cloud.google.com/blog/products/ai-machine-learning/ai-in-depth-serving-a-pytorch-text-classifier-on-ai-platform-serving-using-custom-online-prediction)\n",
    "- [Game of Thrones Twitter Sentiment with Google Cloud Platform and Keras blog post](https://towardsdatascience.com/game-of-thrones-twitter-sentiment-with-keras-apache-beam-bigquery-and-pubsub-382a770f6583)\n",
    "- [AI Platform Pipelines](https://github.com/kubeflow/pipelines/blob/master/samples/core/ai_platform/ai_platform.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
